Skip to main content

Wide Crawl started March 2011

Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.

What’s in the data set:

Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069

The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.

However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.

We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.

If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.

8,528
RESULTS
rss


PART OF
Worldwide Web Crawls
Internet Archive Web Crawls
Media Type
8,528
web
Year
8,528
2011
Topics & Subjects
8,528
crawldata
Collection
Creator
8,528
internet archive
SHOW DETAILS
up-solid down-solid
eye
Title
Date Archived
Creator
Wide Crawl started March 2011
web
eye 454,614
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl415.us.archive.org:wide from Sun Aug 14 03:13:23 PDT 2011 to Sat Aug 13 22:20:18 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 271,613
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl410.us.archive.org:wide from Fri May 13 21:19:40 PDT 2011 to Fri May 13 22:10:12 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 268,758
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl448.us.archive.org:argov from Wed Aug 24 01:32:00 PDT 2011 to Thu Sep 1 12:02:59 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 243,914
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl423.us.archive.org:wide from Wed Apr 27 01:17:43 PDT 2011 to Tue Apr 26 20:39:17 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 228,363
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl420.us.archive.org:wide from Tue Apr 26 17:43:13 PDT 2011 to Tue Apr 26 12:42:46 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 227,186
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl416.us.archive.org:wide from Thu Jun 23 21:11:43 PDT 2011 to Thu Jun 23 14:53:15 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 209,482
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl426.us.archive.org:wide from Wed May 11 12:22:46 PDT 2011 to Wed May 11 08:39:50 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 205,927
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl417.us.archive.org:wide from Sat Apr 30 16:45:54 PDT 2011 to Sat Apr 30 21:10:35 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 189,324
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl426.us.archive.org:wide from Sun May 1 01:24:09 PDT 2011 to Sat Apr 30 22:56:48 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 186,037
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Thu Jun 23 01:41:07 PDT 2011 to Wed Jun 22 19:53:32 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 184,572
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl421.us.archive.org:wide from Sat Jul 16 07:24:18 PDT 2011 to Sat Jul 16 00:57:31 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 172,619
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl422.us.archive.org:wide from Fri Apr 29 00:28:27 PDT 2011 to Thu Apr 28 20:15:25 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 164,874
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl421.us.archive.org:wide from Thu Oct 6 05:28:56 PDT 2011 to Thu Oct 6 00:58:42 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 163,179
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl421.us.archive.org:wide from Tue Apr 26 01:41:22 PDT 2011 to Mon Apr 25 20:29:54 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 160,679
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl416.us.archive.org:wide from Sun Jul 3 19:45:18 PDT 2011 to Sun Jul 3 13:40:14 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 157,053
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl410.us.archive.org:wide from Thu Apr 21 18:29:23 PDT 2011 to Thu Apr 21 17:57:54 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 155,287
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Thu Jun 23 03:05:40 PDT 2011 to Wed Jun 22 20:25:57 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 155,038
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl426.us.archive.org:wide from Sat Apr 30 08:13:45 PDT 2011 to Sat Apr 30 06:46:45 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 153,228
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl424.us.archive.org:wide from Fri Apr 29 18:33:24 PDT 2011 to Fri Apr 29 14:42:40 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 152,091
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl425.us.archive.org:wide from Fri Jul 22 05:35:21 PDT 2011 to Thu Jul 21 23:36:33 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 151,840
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl418.us.archive.org:wide from Thu May 12 19:16:19 PDT 2011 to Thu May 12 17:08:02 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 151,754
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl448.us.archive.org:argov from Wed Aug 24 22:48:30 PDT 2011 to Thu Sep 1 12:13:09 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 150,095
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl425.us.archive.org:wide from Fri Apr 29 18:34:30 PDT 2011 to Fri Apr 29 14:28:25 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 145,849
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl422.us.archive.org:wide from Tue May 10 22:21:14 PDT 2011 to Thu May 12 10:52:29 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 143,108
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl429.us.archive.org:wide from Sat Apr 30 12:31:45 PDT 2011 to Sat Apr 30 11:48:18 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 142,003
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Fri Jun 24 05:16:06 PDT 2011 to Thu Jun 23 22:45:10 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 140,816
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl426.us.archive.org:wide from Sun May 1 05:56:48 PDT 2011 to Sun May 1 03:35:51 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 135,844
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl411.us.archive.org:wide from Mon Jun 6 22:08:59 PDT 2011 to Mon Jun 6 16:02:13 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 135,210
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl425.us.archive.org:wide from Thu Jul 21 18:13:35 PDT 2011 to Thu Jul 21 12:04:47 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 134,650
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl419.us.archive.org:wide from Thu May 12 20:32:45 PDT 2011 to Thu May 12 16:18:38 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 133,937
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl338.us.archive.org:wide from Wed Mar 9 00:51:25 PST 2011 to Tue Mar 8 19:52:22 PST 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 127,577
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl416.us.archive.org:wide from Thu Jun 23 20:04:13 PDT 2011 to Thu Jun 23 14:00:57 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 126,490
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Fri Jun 24 07:39:04 PDT 2011 to Fri Jun 24 01:20:52 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 125,427
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl427.us.archive.org:wide from Sat Aug 27 21:31:42 PDT 2011 to Sat Aug 27 22:07:22 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 120,990
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl415.us.archive.org:wide from Mon Aug 8 22:37:24 PDT 2011 to Mon Aug 8 19:58:57 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 120,039
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl428.us.archive.org:wide from Wed May 11 05:57:13 PDT 2011 to Wed May 11 02:09:02 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 119,204
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl413.us.archive.org:wide from Thu Apr 21 18:53:32 PDT 2011 to Fri Apr 22 04:21:25 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 114,205
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl416.us.archive.org:wide from Sat Apr 30 16:46:07 PDT 2011 to Sat Apr 30 15:17:39 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 112,829
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl413.us.archive.org:wide from Wed Jul 13 12:01:00 UTC 2011 to Wed Jul 13 12:57:54 UTC 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 112,821
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl425.us.archive.org:wide from Wed Jun 15 01:13:49 PDT 2011 to Tue Jun 14 20:47:44 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 112,058
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl425.us.archive.org:wide from Wed Jul 27 22:51:50 PDT 2011 to Wed Jul 27 16:59:32 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 112,004
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl418.us.archive.org:wide from Fri May 13 00:44:55 PDT 2011 to Fri May 13 00:41:56 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 111,383
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl425.us.archive.org:wide from Tue Jul 26 18:06:41 PDT 2011 to Tue Jul 26 12:55:43 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 110,743
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl427.us.archive.org:wide from Sat Apr 30 08:14:44 PDT 2011 to Sat Apr 30 06:19:55 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 109,625
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl414.us.archive.org:wide from Sat Apr 23 00:19:37 PDT 2011 to Sat Apr 23 04:18:58 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 109,001
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl423.us.archive.org:wide from Sat Jul 16 06:47:12 PDT 2011 to Sat Jul 16 00:46:51 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 108,658
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl415.us.archive.org:wide from Wed Apr 20 01:03:39 PDT 2011 to Thu Apr 21 22:48:08 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 108,344
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl420.us.archive.org:wide from Wed Apr 27 07:58:43 PDT 2011 to Tue Jun 14 13:40:31 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 107,872
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl416.us.archive.org:wide from Fri Jun 24 01:53:49 PDT 2011 to Thu Jun 23 19:38:20 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 107,392
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl421.us.archive.org:wide from Tue Oct 4 22:00:11 PDT 2011 to Tue Oct 4 16:07:45 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 107,322
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Fri Jun 24 09:08:29 PDT 2011 to Fri Jun 24 02:59:41 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 106,755
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Fri Jun 24 04:41:37 PDT 2011 to Thu Jun 23 22:14:12 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 104,986
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl411.us.archive.org:wide from Tue Apr 19 23:14:47 PDT 2011 to Thu Apr 21 11:32:30 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 104,560
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl411.us.archive.org:wide from Tue Jun 14 06:14:59 PDT 2011 to Tue Jun 14 00:23:44 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 104,428
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl413.us.archive.org:wide from Sun Jul 3 05:22:17 PDT 2011 to Sat Jul 2 23:10:34 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 103,146
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl421.us.archive.org:wide from Fri Apr 29 00:28:01 PDT 2011 to Fri Apr 29 01:47:45 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 103,122
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl426.us.archive.org:wide from Tue Jul 26 15:28:09 PDT 2011 to Tue Jul 26 09:53:32 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 102,888
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl422.us.archive.org:wide from Fri Apr 29 03:28:27 PDT 2011 to Thu Apr 28 21:48:35 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 102,825
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl428.us.archive.org:wide from Wed May 11 15:59:57 PDT 2011 to Wed May 11 13:06:59 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 101,789
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl417.us.archive.org:wide from Sun May 1 07:36:45 PDT 2011 to Sun May 1 03:33:21 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 101,700
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl426.us.archive.org:wide from Sat Apr 30 18:51:58 PDT 2011 to Sat Apr 30 17:48:25 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 100,525
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl411.us.archive.org:wide from Sat Aug 27 10:49:07 PDT 2011 to Sat Aug 27 05:40:30 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 100,323
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl448.us.archive.org:argov from Wed Aug 24 23:49:34 PDT 2011 to Thu Sep 1 17:17:29 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 99,481
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl423.us.archive.org:wide from Sat Jul 16 09:12:32 PDT 2011 to Sat Jul 16 03:34:11 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 97,204
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Thu Jun 23 16:46:29 PDT 2011 to Thu Jun 23 10:34:07 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 96,498
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl424.us.archive.org:wide from Sat Apr 30 00:32:31 PDT 2011 to Fri Apr 29 19:53:45 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 96,124
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl417.us.archive.org:wide from Tue Aug 9 22:27:01 PDT 2011 to Tue Aug 9 20:55:46 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 94,438
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Thu Jun 23 03:26:43 PDT 2011 to Wed Jun 22 20:46:48 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 94,366
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl423.us.archive.org:wide from Wed Jun 15 03:06:30 PDT 2011 to Tue Jun 14 21:53:14 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 94,231
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl412.us.archive.org:wide from Tue Sep 27 13:48:57 PDT 2011 to Tue Sep 27 07:46:21 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 93,329
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl421.us.archive.org:wide from Thu Oct 6 02:51:02 PDT 2011 to Wed Oct 5 21:45:35 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 93,286
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl418.us.archive.org:wide from Fri May 13 15:51:53 PDT 2011 to Fri May 13 19:08:43 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 92,847
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl413.us.archive.org:wide from Sat Sep 24 16:30:43 PDT 2011 to Sat Sep 24 10:21:46 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 92,815
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl420.us.archive.org:wide from Tue May 10 06:59:01 PDT 2011 to Tue May 10 02:36:49 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 92,463
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl429.us.archive.org:wide from Sat Aug 27 10:03:44 PDT 2011 to Sat Aug 27 04:21:29 PDT 2011.
Topic: crawldata